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ABSTRACT 

A self-paced, individualized, multimedia physics 
course developed by the" New York Institute of Technology for the U. 

5. Naval Academy with funds provided by the U« S. Office of Education 
is evaluated. Student performance is measured by "core problems." A 
core problem is a terminal objective in problem form. The student 
uses his results as a basis for choosing one of three alternatives: 
to proceed to the next core problem, to try the core prime problem {a 
variation of the core problem) , or to address a sequence of enr' ling 
problems (illustrating major steps for the achievement of the core 
problem). In comparisons with "conventional" instruction, no 
achievement differences were found between the forms of instruction. 
However, reasons were cited which indicated that under the conditions 
of the experiment the self-paced students were placed at a 
disadvantage with respect to these criteria. It was concluded that 
the individualized, self-paced system makes a substantial difference 
in performance over conventional instruction. Included are 
evaluations of the media used and the cost effectiveness of the 
program. Extensive tables of statistical data are added as 
appendices. (Author/TS) 
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Introduction 



The final revision of the Self-Paced Physics program was character- 
ized by extensive alterations. These changes were arrived at through a 
systems approach using data, experience, and judgment. Our purpose here 
is to ascertain whether the revision process worked and, to a lesser 
extent , to compare the program with conventional instruction. 

It has only been possible to obtain gross measures of program 
effectiveness since an evaluation of the installed, free-running system 
was not planned as part of the study. Of course, the need for evaluating 
the final revision could not have been anticipated at the inception, 
just as the many difficulties and extensive alterations were unexpected. 

Although circumstances limited the possible analysis, we can state 
with some assurance that the revision caused an increase of over fifty 
percent in students reaching the criteria. Comparisons with conventional 
instruction are much less clear, but it appears that the system holds a 
substantial margin in performance on the course objectives. Before 
discussing the evidence for these conclusions, it is helpful to summarize 
some of the features of the last revision. 



Revision Features 

A central feature of the self-paced program is its responsiveness to 
immediate past performance. In the latest version this is accomplished 
primarily through performance on a core problem (a terminal objective In 
problem form) . The student uses this as a basis for choosing one of 
three alternatives: to proceed to the next core problem, to try the 

core prime problem (a variation of the core problem) , or to address a 
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sequence of enabling problems (illustrating major steps for the achieve 
ment of the core problem) . 

Thus, when a student is able to solve a core problem with confidence, 
he maximizes his rate by moving directly to the next objective. Some 
Students expect that they can execute problems similar to tha core after 
having seen the correct solution. Often, such students have made a minor 
error in the core and only need a similar problem for practice and rein- 
forcement; this is provided by the core prime problem. When a student 
incorrectly assesses his ability to solve the core prime, or when he 
realizes that his understanding of the core problem is deficient, then 
he takes the enabling problem sequence. The core prime problem is 
always encountered at the end of the enabling sequence. 

In preceding versions of the program, all students worked at all 
problems in ascending difficulty. The primary responsive agents were 
branches to remedial pages in the case of errors. The substantial re- 
visions were designed to economize on students study time, to provide 
advance organizers in the form of core problems, to use core prime 
problems as an immediate "posttest" for reinforcement and practice, to 
avoid emphasis on "second-guess remediation" which was unsuccessful in 
anticipating which one of myriad possible errors was likely to be com- 
mitted, and generally to increase the possible number of learning paths 
for greater individualization and motivation. These are detailed in 
the. Course Development Report and the Revision Process Report. 

Other revisions were made in accord with the systems approach. 
Information Panels were created to orient students and provide over- 
views, Additional problems were created for "trouble topics." 




2 

5 



Problem statements were revised to avoid any ambiguities, and problems 
which did not adequately support objectives were removed. A Study Guide 
was created to serve as the primary agent of internal management: it 

reveals answers to the student and directs his progress through alterna- 
tive paths on the basis of these answers. By using Study Guides and a 
revised format for the Problem and Sol *:ion Books, the student was re- 
lieved of a great deal of page turning. These and many additional revi- 
sion steps are presented in the Revision Process Report. 

Revision Evaluation 

The data available for making an evaluation of the revised program 
were extremely limited. Provisions were not made for data collection under 
controlled conditions because the delivered version was a free-running 
‘ course with students progressing at their own rates and using the materials 
they most preferred — hardly an Ideal experimental situation. 

It was not possible to compare the Fall 1969 and Fall 1970 runs on 
the basis of test performance because the Academy faculty chose a new 
instrument for class grades (diagnostic testa) and constructed a new 
Final Examination, not at all parallel to the prior examination. The 
Academy was more interested in evaluation measures for midshipmen than 
for the program, as they should be. Even posttests were changed in con- 
tent and level, and from a mandatory grade-establishing status to an 
optional progress check. 

Only one "test" collection remained sufficiently unaltered and was 
administered under sufficiently similar conditions— problems from the 
Problem and Solution Books. Even a majority of these problems have been 
changed in format from multiple-choice to completion and constructed re- 
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sponse, Tli is severely narrowed our potential for comparison because 
multiple-choice answers may be guessed, unlike constructed responses. 
Statistical corrections for guessing are inadequate due to the fact that 
many students will work until they match one of the choices, although 
constructed responses do not provide similar feedback. Moreover, the 
answer solutions are not equally probable. 

There is special merit to comparing core problem performance before 
and after revision. The core problems represent terminal objectives and 
as such, performance on these problems is the prime measure o achieve- 
ment of course objectives. A straightforward comparison of core problem 
however, is unwarranted because in older versions they appeared at the 
end of an enabling sequence, whereas now they appear before any other 
problems on that objective. Essentially, the core prime has taken the 
’ place of the core as the final exercise on the given objective. 

Consequently, the appropriate comparison is between Fall 1969 core 
problems and Fall 1970 core prime problems. We found the subset of 
core primes which are exceptionally parallel to the original cores 
(usually only number changes) , which are multiple choice with very 
parallel disu actors, and which have always had the same mandatory 
status . 

Fourteen pairs of cores and core primes satisfied all the required 
constraints. The results are listed in Table 1. Assuming equivalent 
student populations, a S7% increase in achieving criteria has been 
accomplished by the revised system over the 1969 version. Although 
the number of matched problem pairs is small, they are rather repre- 
sentative and well-distributed. This, and the rather uniform improve- 
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merit throughout the entire list, strongly suggests that the result is a 
good gross measure of the efficacy of the revised approach. 

In recording the core prime results, any questionable response was 
counted as incorrect. When a student successfully completed a core 
problem, and proceeded to the next objective, the core prime was scored 
as correct. 

Missing responses are not included in the calculations for compari- 
son purposes because the 1969 computations did not include rnisr»j.ng re 
sponses. Table 2 shows the 1970 data adjusted to treat missing responses 
as incorrect. This provides a more meaningful absolute measure of the 
percent reaching criterion, 72%, as opposed to 77% for comparative use. 

Ideally, it would have been desirable to perform statistical analysis 
on student populations in various background variable. categories . This 
was not possible because individual data by student by problem in the 
Problem and Solution Book were unavailable for the 1969 run. The assump- 
tion of equivalent populations, however, is reasonably well supported by 
a comparison of background variables averaged over the whole group. 

Comparison With Conventional Instruction 

t ny assessment of "conventional" instruction is difficult because of 
the large variability in the person of the instructor; the classes of two 
different lecturers may perform quite differently. A comparison of the 
self-paced program with its conventional counterpart is further obfuscated, 
as we shall see, by several factors pertaining to the common Final 
Examination . 

One hundred self-paced students and 114 conventional students of 
Physics SP211 took the uniform Final Examination in Fall 1970. With the 
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New 

Core 


TABLE 

PROBLEM NUMBER 


1 

PERCENT 


CORRECT 


New 

Prime 


Old 

Core 


1970 

Core 


1969 

Core 


1/6 


1/9 


1/16 


79 


46 


2/1 


2/5. 


1/15 


92 


60 


4/6 


4/10 


4/8 


90 


49 


4/16 


4/20 


4/21 


85 


42 


5/13 


5/17 


5/10 


59 


51 


7/5 


7/8 


7/9 


80 


29 


7/10 


7/14 


7/24 


64 


41 


7/27 


7/29 


7/22 


58 


43 


8/5 


8/8 


8/11 


65 


52 


8/13 


8/17 


8/17 


68 


52 


9/1 


9/3 


9/4 


86 


76 


11/1 


11/4 


11/1 


92 


53 


11/15 


11/17 


11/24 


93 


79 


13/11 


13/14 


13/19 


66 


15 






Average 


■ 77 


49 
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PROBLEM NUMBER 



PERCENT CORRECT 



tabljLJ. 



New 

Core 


New 

Prime 


Old 

Core 


With 

Omissions 


Without 

Omissions 


1/6 


1/9 


1/16 


79 


79 


2/1 


2/5 


1/15 


91 


92 


4/6 


4/10 


4/8 


89 


90 


4/16 


4/20 


4/21 


83 


85 


5/13 


5/17 


5/10 


49 


59 


7/5 


7/8 


7/9 


77 


80 


7/10 


7/14 


7/24 


59 


64 


7/27 


7/29 


7/22 


48 


58 


8/3 


8/8 


8/11 


54 


65 


8/13 


8/17 


8/17 


56 


68 


9/1 


9/3 


9/4 


73 


86 


11/1 


11/4 


11/1 


92 


92 


11/15 


11/17 


11/24 


92 


93 


13/11 


13/14 


13/19 


64 


66 



Average 72 77 
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omission of. problems which were ambiguous or not related to NYIT course 
objectives (see Appendixes B and C for full details), both groups attained 
55% averages , 

Course designers point to several circumstances which may have pre- 
vented the self-paced group from a stronger relative performance. First, 
the total performance is very poor for all students, especially for a 
mu 1 1 i p le - ch o .1 ce test on which random choices would score 20%. One may 
be immediately suspicious that the examination did not test what it in- 
tended to test, even though our specialists agree that most of the items 
are good problems. 

If the test items were fair, then it is likely that the test condi- 
tions were different than those implied by the course objectives. Indeed, 
jor an entire semester students were allowed to achieve objectives at 
their own rate, whereas it was demanded that they complete 55 problems • 
in 3 hours on the Final Examination. Most of these problems required 
computation and contemplation; it took an investigator an hour just 
to read the problems with enough care to classify them. These considera- 
tions suggest that the Final Examination performance is, in large measure, 
the result of a race for which neither conventional nor self-paced stu- 
dents were trained. 

The Final Exam was developed by concensus among the Academy faculty 
members. Surely no instructor would agree to the inclusion of topics which 
were not covered in class. This, coupled with the fact that only a frac- 
tion of the objectives was sampled, gave an advantage to those student 
sections which covered the least material. We know from the study guide 
responses that the self-paced sections covered all objectives. Thus, any 
conventional sections which were allowed to omit material due to its relative 
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''unimportance" would be able to concentrate more intensively on less 
maerial. 

Self-paced students suffered other disadvantages on the Final 
Examination. The objectives which they worked toward in the Problem 
and Solution Book a *e primarily multiple-step (see Report TR 5,8 for 
details of this classification scheme) • The Final Examination items, 
however, are primarily one— step. The correlation between one— step and 
multiple— s tep problems is ,65 (TR 5,8) ; rot high enough to imply that 
a student who is performing well in one category will perform equally 
well In the other. 

Moreover, the diagnostic tests given to the self-paced group during 
the semester (for grading purposes) must have adversely influenc d the 
students. These problems were primarily zero-step. Certainly this 
caused a poor state of preparedness for the Final because zero-step /one- 
step correlation is only ,44, Thus, the Final Examination tested most 
in the category with which self-paced students had the least exposure. 

Finally, we note that the Final Examination was constructed before 
the class reviews , The classroom lecturer could then emphasize principles 
and methods which were relevant to the Final, (We do not know whether this 
was actually done). This practice is perfectly proper, and even desirable, 
but the self-paced group could not enjoy a similarly well— directed review, 

it ’ is been suggested that the Final Examination has not tested what 
was taught, that the self-paced group may have experienced some disadvan- 
tages, and that the conventional group may have enjoyed some benefits. 
Nevertheless, the self-paced and conventional groups had the same perfor- 
mance, The only hard conclusion which can be drawn from these rather 



insubstantial considerations is that self paced instruction is no less 
effective than conventional instruction. 

Perhaps a more meaningful comparison can be made. Before he en- 
counters the core problem, a self— pa^sd student receives instruction 
which , in content and delivery, is virtually identical to conventional 
instruction , That is, he reads the textbook assignment, is exposed to 
vidsotaps instruction or another medium which essentially replaces a 
lecture demonstration, and is informed through an Information Panel of 
his terminal objective (a good conventional section should Include thi 
feature, as the Academy conventional sections did). 

The self-paced student, therefore, has had essentially "conventional 
instruction,, albeit automated, when he addresses the core problem. The 
core problem corresponds to a homework problem in the textbook in this 
view. Self-pacing, as a responsive dynamic system, really begins when 
the result of the core performance is known. Various media options and 
self-paced individualized paths through a learning sequence start here. 
The core prime problem ends the sequence for most students (excepting 
those few needing tutorial assistance) , 

Consequently a comparison of conventional and self-paced instruc- 
tion is provided by comparing performance on very parallel core and core 
prime problems for the same students. This tack certainly removes any 
need to assure homogeneity of experimental and control groups. The 
results for the highly parallel problems are recorded in Table 3 (omis- 
sions are scored as incorrect in Table 3) . Here we see that 16% more 
students reach criterion through the program, an increase of 28% over 
the "conventional" performance, 

A criticism of this analysis may be raised that we are only com- 
paring less instruction with more instruction and finding in favor of 



Me know from our first reislon 



the latter. This is not at all true, 

(which expanded all materials prior to "core”) that simply expanding 
conventional instruction on an objective seldom causes a significant 
increase in performance (see Introduction of Course Development Report) . 
Although a more definitive test is desirable, we believe that this analysis 
shows the individualized, self-paced system makes a substantial difference 
in performance over conventional instruction. 
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TABLE 3 



New 

Core 


PROBLEM NUMBER 




PERCENT 


CORRECT 


New 

Prime 


Old 

Core 


1970 

Core 


1970 
Pr line 


1/6 


1/9 


1/16 


60 


79 


2/1 


2/5 


1/15 


67 


91 


4/6 


4/10 


4/8 


73 


89 


4/16 


4/20 


4/21 


44 


83 


5/13 


5/17 


5/10 


40 


49 


7/5 


7/8 


7/9 


60 


^ '» 


7/10 


7/14 


7/24 


35 


59 


7/27 


7/29 


7/22 


31 


48 


8/5 


8/8 


8/11 


56 


54 


8/13 


8/17 


8/17 


36 


56 


9/1 


9/3 


9/4 . 


75 


73 


11/1 


11/4 


11/1 


68 


92 


11/15 


11/17 


11/24 


92 


92 


13/11 


13/14 


13/19 


46 


64 



56 72 
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Other Evaluation Measures 



Performance is the most obvious measure of instructional value , 
but other important considerations should also be addressed. Some 
evaluation may ha made of the students 1 time required for study and 
attitude toward the course. 

The source of th_s evaluation is a questionnaire developed and 
distributed by the Academy at the conclusion of the Fall semester 1970 
(see appendix D) , It is unfortunate for purposes of comparison that 
most of the questions were not distributed to conventional sections. 
Despite this and other shortcomings of the questionnaire, the responses 
of the self -paced group provide a general picture of student study time. 



attitude and effort. 

Student attitude regarding the time— effectiveness of the program 



is reflected in several of the questions. 

As a result of s Ludent-vacing 3 the amount of time 
I was able to spend on other subjects was: 

A • Touch more »•**•••«•••»••• * 

B * more ■#•#*»••• 35 

Cm about the scone * • * 20 

T ) . less ##**»***»*#•*■••»■••********■•*•*•• 20 
j? , much less • * ••••■*•*#•*•*•»«***•*» * * * * ^ 

Almost half of the students thought they were spending less time because 



of self-pacing. If is probably true that self-paced students spent less 
time in study than the conventional sections, but the actual time data 
was too sparsely recorded (by the students themselves) to verify or deny 



this* One question shows a rather modest homework effort on the part 
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of self-paced students: 



Did. you study physics outside the classroom? 



A , over 6 hrefweek . 12 

B t 3-6 hrs per tjeek. ............ SB 

C\ 1-5 Tzrs per week, 5*5 

fl, seldom, . . 6 

E w n* e *o e z* ^ ^ 



A less personalised question reveals that the course work is of 
manageable proportions : 

Tfte amount of material covered in the course 



should be: 

/I. greatly increased , * * «. * * 0 

13 . e a.s e d * 2 

C . ?Min£a£ne<5 * . £5 

£>, decreased 1? 

E . greatly decreased 0 



A very important fact is evident. Quite unlike t o 1969 experi- 
mental run 5 students taking the revised program had no load placed 

on their study time. The principal goal of the 1970 revision T as 
successful . 

Student attitude toward the program is less well defined* Overall , 
the course was considered more difficult than other science courses and 
half of the students* felt they learned less from this "method of material 
presentation" as compared with conventional instruction. More detailed 
questions regarding self— pacing and various components 5 however, indicate 
a general preference in favor of the program's approach. 
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It is part of academic folklore that, among all the basic subjects 
In the engineering curriculum, the introductory course in physics with 
calculus is by far the most difficult. In view of this notoriety, the 
self-paced students’ attitude about the difficulty as reflected in the 
questionnaire is not extreme* 

Compared to other science classes that I have taken 
at the Academy I think this class is 

A # YftlA Oh easier #■■**•***»••**•*»**•*■•••••* ^ 

Em 0ClS*Zr&O^ • • *••■•***••■•■•••***•«*•*•■***** ^ 

C. about the same 30 

Dm harder * •■■***#*##»*»***»*****»***» < »** 9 32 
E * much hai -cter m #*#*•••#•••*»***•■*.******•• 3,0 
No respons e *■ #i i # B#*«#*«-«**««««^*»**^^*^***‘ (3) 

Only 10 students out of 81 answering the question thought the course was 



much more difficult than other science courses. 

Another question reflects the students f attitude about what they 

think, they learned: 

Because of the method of material presentation 
compared to the v conventional ff method of lecturing 
I believe I learned: 

A * much more 0 

B . more ...» 20 

about the same ■■•••■••**•***■*•******•■ 20 

D 9 leSS + 9»»*9mmmm9mm*»U*99999*******99** m * 2 B 

E 9 TTtuch less .,,****#■***••••■********•***• 32 
No response ■ ••••••••****-•****••*•**•**■*** C3) 
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Of course, this response may show a "protest vote" by the students who 
felt that the course was harder than other science classes . There is 
almost complete overlap between the students responding D and E here 
and those responding D and E to the preceding question. 

Some of the questions and responses regarding component of the 
program are listed below A generally positive attitude toward s e 1 f “ 
pacing 5 laboratories, the Problem and Solution Book, and various media 
(excepting TV tapes) is reflected here, 

j~ have all physios tabs 

in the self -paced format. 

A * q trong ly taou Z d It'ke *******■»*•»■■»**■■• 45 

B, would rather . * , * < * * 36 

C . no preference , * * * * 5 

Zh would rather not * * 6 

Em would not want to m *••••••••*•••••••••«« 0 

As a way of organizing a course 3 I think that letting 
students pace themselves and manage their own pro- 
gress in : 

A m extremely beneficial 5 

B m very It e“ne ###*■■#*.#****#*••••*'•*• 50 

C a adequate •■••**■■*••••**»«»****»■•*•* * * 34 

Dm not very beneficial . 22 

* 
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Were the questions and information in the Problem 
and Solution Books helpful in understanding the 
objectives? 

A * extremely helpful - * * 4 

B, very helpful . * . . - 26 

Cm adequate ******* ##•■*•*■* #*****###*■#•• 32 
D 9 not very he Ipful • •*#**•**«****■*•*»**#*« IS 
E • us e l es ©•*****•*#«*#**•***#####•*■###••• 2 

Did you find the Problem and Solution Book helpful 
in preparing for Quarterly Diagnostic Tests? 



A* i>ery valuable * * * . * 11 

B* helpful . . „ . . . 33 

C * so-so * * * , * * * * 23 

D. not very helpful 10 

E * - wasted effort S 



Did the Problem and Solution Book provide adequate 
preparation for the Progress Checks? 

A. very good preparation 3 

B * good preparation * * , , * * 25 

C * adequate preparation ****«•««. 39 

D* poor preparation * ************ * • * * * 13 

£7. poor preparation 2 
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Was the material -presented cn the TV tapes 



adequately explained? 

A. very clear. , , 2 

B. clear IS 



C. adequately clear 43 

D, poorly explained. 11 



E. unusedb le 5 

No response (6) 



Did seeing the physics phenomenon demonstrated on 
TV aid in your unders banding of the concept? 

A, extremely helpful 2 



B. very helpful • 8 

C. adequate 40 

D. not very helpful 22 

E. ' useless 4 



^ ................................ i 8 1 

Did you study physics outside the classroom? 

A. over 6 hours per week. 11 

B. 3-6 hours per week 32 

C. 1—3 hoTA^rs per ^weeh.. 33 

D . s e Idem 8 
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The use of " Talking Books " and the audio tapes was 
A . helpful, 5 

5 • £3 L pfbL i i • • • • ■ | l f I : 3 i i • i « i > • f 25 

* adepU’Ctte * * • * •**•••••■■•*■• * * * • »••*■*••• 37 

D . ^^ 2 ^ helpful . 23 

£*, i^ssZ'SSS ^ 

Ho respons e • • , •«. ».***#. ***»«.. ••«••»•*•*** Cl) 

Using the "Illustrated Book 11 is 
A . x^ahaecJo te ,»••*»** *•*****,»* , ,,**,*••*•• • 5 

B * he tpful ...««».****#♦*•*•••••**.•*•-•**• 25 

C* OK 55 

£>. not very helpful, * 2 6‘ 

S’. boated effort ? 

Ho ises‘pon,s C 3) 

Students ranked the media they found most effective for learning 
physics in the following order (from most to least effective).* 

Problem and Solution Book 

Instructor 

Text 

Talking Book/ Illus trated Book 
TV Tape 

This list descends from the medium which requires the most active 
student participation to that which allows the most p salve response. 

On this basis s we are inclined to agree with the students 1 evaluation. 
Effectiveness does appear to be closely related to active participation 
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in less complex courses this participation is adequately measured by 
response demand frequency* 

The full questionnaire with responses is reproduced in Appendix B. 
Media Evaluation 

One of the goals set forth at the inception of the program was 
to establish optimal individualization with respect to media. We 
sought to determine what media were most efficacious for a student 
with a known set of background variables. The term media here refers 
to two classes of material: audiovisual components, which include 

videotape, talking book, illustrated text, and lecture/demonstration; 
and fundamental learning instruments, which are the Problem and Solu- 
tion Book, the textbook, and the on-site instructor. 

We were unable to detect significant differences in performance 
due to the audiovisual materials. However, individual students do 
have preferences among audiovisual components and this is the basis 
of the present system of voluntary selection. 

Although the use of audiovisual materials evidenced no significant 
performance differences in the operating program (see 5*0, Statistics: 
1969 Experiment) , we must not conclude that they are without signifi- 
cant value. We believe these materials are useful for teaching 
"trouble topics" which are often difficult for students to concep- 
tualize, These often require visual or, auditory amplification, re- 
dundancy and repetition in various perspectives, and motion. Trouble 
topics had little representation on exarinations even though some 
instructors insisted that they were essential to the course content. 
This disparity between what a student ought to know (as revealed in 
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course objectives) and whet he is expected to know (as revealed in 
testing) is apparently widespread in the community of science educators. 
For many professors, it may be that conscience dictates the inclusion 
of some difficult topics in the course-- later, conscience dictates that 
these topics be excluded from examinations. 

Audiovisual materials are also used to provide overviews s enrich- 
ment, and to demonstrate the relevance of subject matter in real-world 
settings and real=life applications* It seems that these are not signi- 
ficant to the task of achieving the specific behavioral objectives. 
However , such presentations are likely to stimulate audience interest 
in the subject matter — a very desirable general objective for any 
course. It is improbable that audiovisuals increased interest or 
motivation in Academy midship r they have no time for such luxuries* 
Other less motivated and less goal-directed student groups, however, 
may profit more from audiovisuals. 

Three distinct weaknesses of audiovisual materials for a majority 
of college physics topics are now evident: they are and 

time-consuming sources of information (as compared with textbooks , for 
example) ; they allow for student interaction only with some difficulty 
and awkwardness (especially notable for high level objectives) ; and 
they are not easy to M skim, ,? so that locating a particular fact or objec- 
tive may require extensive searching. These weaknesses suggest that 
audiovisuals should not be recommended casually for run-of-the-mill 
obj actives— even when cost is not an important consideration* 

The Problem and Solution Book, the textbook, and the on-site 
instructor are indispensable components of the program as it is presently 
constituted. Student opinion rated the Problem and Solution Book as the 

21 




24 



most effective learning medium. The substantial performance gains 
deriving from the Problem and Solution Book and the favorable student 
reaction to it strongly support this opinion* Indeed* our investigators 
now regard the invention of the Problem and Solution Book as the most 



Students rated M ins tractor n as the second most effective medium. 

The questionnaire does not specify whether this refers to the on-site 
instructor who is essentially a highly interactive tutor, or to an 
idealized conventional lecturer. It may be that rating the instructor 
as an effective medium was made largely on hypothetical grounds, since 
several oral interviews indicated that few students sought help from 
the instructor. 

A most important finding of our 1970 data analysis (see Appendix E) 
concerns the pivotal role of the textbook. Of the seventeen students 
who thought the textbook was one of the two least effective media, fif- 
teen fell below the average on the adjusted final examination (see 
Appendix B for core related sub test) : 8.6 as compared %?±th the 14,1 

average. Moreover, a similar finding holds for the core and core 
prime problems listed in Table 2: 52% for this group as compared with 

the average of 72%, Two-thirds of the fifteen students who considered 
the textbook ineffective and who fell below average on the core related 
subtest were below average in English Comp, scores. 

The implication is that those students who find the text ineffec- 
tive, or who have difficulty comprehending it, will perform poorly. 

The remedy seems equally clear. Required reading from the textbook 
should be abolished, and this material should be incorporated into the 
Problem and SoIul on Book format. 



important derivative of our development program. 




22 



25 



Cost Effectiveness 



The cost effectiveness of an educational system is often taken as the 
number of criteria attained per unit cost. While this measure has a cer- 
tain appeal, it is an abstraction insofar as existing educational institu= 
tions do not graduate students on the basis of the total attained criteria. 
Thus, a more realistic measure of cost effectiveness is simply the number 
of students processed through the course (regardless of achievement) per 
unit cost. * 

Instructor salaries constitute a major cost in education. Academy 
policy limits class sections to no more than twenty students per instructor 
Although many more students can be served by one instructor using the self- 
paced program, the Academy constraint prevents the realization of a gain 
in cost effectiveness over the conventional system# Estimates of the 
number and cost (in man-hour currency) of instructors and clerks needed 
to serve different numbers of students are given in the cost section of 
TR 5.10, Course Development. Generally, cost effectiveness increases 
with increased numbers of students. In some cases a gain of 100% or more 
in cost effectiveness seems likely with the use of the existing program. 

Another major cost of education is overhead. Through the use of the 
self-paced program, there exists the possibility of reducing the required 
classroom space — a factor which would further enhance the program cost 
effectiveness (see the Course Development report for further discussion 
of this point) . 

For educators who are considering developing their own self-paced 
program, the attractive possibility of a highly cost effective program 
must be amortized against development and installation costs# These 
costs vary greatly, depending upon the materials and sophistication 
desired. Since a similar variability exists in determining the cost 
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effectiveness of an operating program, we suggest that estimates assessing 
specific situations be made with the assistance of the Course Development 
report. 
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APPENDIX A 



1970 CORE AND CORE PRIME DATA 



(Columns refer to Segment/Core 
Problem and Segment/Core Prime 
Problem, respectively.) 
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APPEND’ X B 



PROBLEMS OMITTED FROM FINAL EXAM 



The Fall 1970 Final Examination contained fifty-five problems. Of 
these , twenty-six were associated with objectives and learning materials 
developed by U.S. Naval Academy Faculty. These materials were being 
used for the first time and so the items were not validated. Of the 
remaining twenty-nine, two were found to be ambiguous and two did not 
address core problem objectives. The statements of these problems are 
listed below. 

Problem 1 is ambiguous because distractor B is a correct response 
as is the intended answer D. Problem 7 addresses an objective which was 
removed from the program in the last revision; it was decided to con- 
fine topics in circular motion to uniform circular motion. In Prob- 
lem 26, "none of the above" may be taken as the correct response be- 
cause, as fuel is consumed by the rocket, the mass of the system is 
changing. Self-paced sections were well exposed to variable mass 
aspects of rocket flight as an Academy requirement, but conventional 
sections seemed unattuned to this supposed "catch." This, and the 
failure to sample any variable mass objectives on the Final, leads us 
to suspect that conventional sections in effect bypassed the topic. 
Finally, Problem 36 requires a knowledge of the expression for the 
period of a pendulum. This was not one of the NY T T objectives. 
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Problem 1, The physical quantities of length, mass, and time in the 



MKS 


sys tern 


are : . 


A. 


derived 


quantities , 


B, 


s tandard 


qua ntities , 


C, 


relative 


quantities , 


D. 


fundamental quantities. 


E. 


none of 


the above. 



Problem 7, A particle of mass m moves in circular motion vita a 
constant tangential acceleration of 32 ft/sec 2 . If 
the radius of the path is 8,0 ft,, what is the magnitude 
of the total acceleration when the tangential velocity 
is 16 ft/sec? 

A. 32 ft/sec 2 

B . 64 ft/sec 2 

C. 16 ft/sec 2 
B, 45,3 ft/sec 2 
E, 22.7 ft/sec 2 



Problem 26. A rocket that weighs 5000 lb. on earth is in free space and 
fires a small course correction motor, generating a thrust 
of 60 lb. What acceleration results? 

A. 1,2 x 10 -2 f t/sec 2 

B. 38.4 x 10“ 2 ft/see 2 

C. 1,92 x 10 2 f t/sec 2 

D. 2.4 x 10“ 2 ft/see 2 

E. None of these 
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Problem 36, 



A Simple pendulum with a period of 1 sec o - the surface 
of the earth is carried to a point near the moom where 
the gravitational acceleration is 1.1 m/set . The period 
of the pendulum is: 

A. 1,1 sec 

B. 3.3 sec 

C. 3.0 sec 

D. 9,0 sec 

E. independent of the forces of gravity 
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APPENDIX C 



U. S, NAVAL ACADEMY — FINAL EXAMINATION, FALL SEMESTER 1970 

PHYSICS SP211 
GRAPH ANALYSES 



Graph per Question of Exam: 

1. Total Students of SP211 

2, Comparative Analysis of: 

(1) Conventional Sections of SP211 

(2) Self-Paced Sections of SP211 



in terms of 

— lowest 20% students in raw score performance (L) 

— mid-low 20% students in raw score performance (ML) 

— middle 20% students in raw score performance (M) 

— mid-high 20% students in raw score performance (MH) 

— highest 20% students in raw score performance (H) 
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APPENDIX D 



STUDENT QUESTIONNAIRE 



The questionnaire was devised by Academy 
faculty members, It was given In January 
1971 to students of the Fall '70 semester. 
Eighty- two students who had complete back- 
ground information are included in the 
statistics , 
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Directions: Please answer the following questions on the mark-sense 

card which has your nr me on it. 



1. What did you think of the student-paced laboratory periods? 



2 , 



3, 



4, 



5 , 



A, 


a real experience 


12 




B, 


enjoyable 


50 




C. 


no difference 


10 




D , 


not enough guidance 


5 




E, 


learned little 


5 




How many hours, on the average, did you spend 


per experiment? 




A. 


less than 1 hour 


3 




B * 


1—2 hours 


37 




C, 


2—3 hours 


34 




D. 


3—4 hours 


8 




E * 


over 4 hours 


0 




Were the la 


boratory experiment guides detailed 


enough for you 


to 


understand 


the principles involved? 






A* 


crystal clear 


2 




B. 


clear 


22 




C. 


cloudy in parts 


50 




D. 


confusing 


6 




E, 


didn’t understand the principles 


2 




Were the number of experiments available at any one time adequate? 


A. 


need more up longer 


22 




B, 


adequate 


56 




C, 


need only 1 at a time 


2 




No 


response 


(2) 




I 


have all physics labs in 


the self --paced 


format 


A, 


strongly xjould like 


45 




B. 


would rather 


26 




C, 


no preference 


5 




D, 


would rather not 


6 




E . 


would not want to 


0 





6. Did the Problem and Solution Book provide adequate preparation for 



the Progress Checks ; 

A* very good preparation 3 

B* good preparation 25 

C « adequate preparation 39 

D. poor preparation 13 

E. poor preparation , 2 
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7 . 



The amount of material covered in the course should be: 



A, greatly increased 0 

B, increased 2 

C, . maintained 63 

P. decreased 17 

E. greatly decreased 0 



S . Compared to other science classes that I have taken at the Academy 

I think this class is 



A. much easier * 0 

R. easier 9 

C* about the same 30 

D, harder 32 

E, much harder 10 

No response (1) 



9- Because of the method of material presentation compared to the 
"conventional” method of lecturing I believe 1 learned: 



A* much more 0 

B. more 20 

C. about the same 20 

D. less 29 

E. much less 12 

No response (1) 



10. To what degree were the mechanical steps (i.e*. the magic pen* 
flipping pages., using the video tape recorder) a hindrance to 
your learning the material 



A. 


a great hindrance 


6 


B. 


a hindrance 


20 


C. 


no hindrance 


32 


D. 


added some variety 


20 


E. 


actually enjoyable 


4 



11 . As a way of organizing a course , I think that letting students 
pace themselves and manage their own progress is : 



A. 


extremely beneficial 


3 


B, 


very beneficial 


30 


C. 


adequate 


24 


B. 


not very beneficial 


22 


E. 


useless 


3 
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If X were to take another course in science or math s I would 
prefer student paning to be: 



A. completely free 2 

B. less restrained 10 

C. the same 34 

D. more restricted 24 

E. abolished ' 12 

X believe that X was able to schedule my own time: 

A* very profitably 6 

B. profitably 12 

C, adequately 28 

B. less profitably 30 

E. very poorly 6 



As a result of s tudent^ pacing ? the amount of time X was able 
to spend on ether subjects was: 



A * 


much more 


4 


B * 


more 


35 


C. 


about the same 


20 


D. 


less 


20 


E. 


much less 


■ 3 



Were the questions and information in the Problem and Solution 
Books helpful in unders tanding the objectives? 





A. 


extremely helpful 


4 




B, 


very helpful 


26 




e. 


adequate 


32 




D- 


not very helpful 


18 




E. 


useless 


2 


Did 


the questions in the Progress Checks require 


a greater or 


lesser degree of understanding than yoi expected 


on the basis 


of 


the stated objectives in the Study Guide? 






A. 


much greater understanding 


13 




B. 


slightly greater 


42 




C, 


exactly the same 


20 




D * 


slightly less 


6 




E. 


much less 


1 


Did 


you find 


the Problem and Solution Book helpful in preparin; 


for 


Quarterly Diagnostic Tests? 


■ 




A. 


very valuable 


11 




B. 


helpful 


33 




C. 


so-so 


23 




D. 


not very helpful 


10 




E. 


wasted effort 


5 



18 



Was the material presented on the TV tapes adequately explained? 



A, very clear 

B, clear 

C, adequately clear 
Bp poorly explained 
E. unuseable 



No response 



2 

15 

43 

11 

5 

< 6 ) 



19 , Bid seeing the physics phenomenon demonstrated on TV aid in your 
understanding of the concept? 



A, extremely helpful 2 

Bp very helpful 8 

Cp adequate 40 

D, not very helpful 22 

E, useless 4 

No response (6) 

20 . X made use of the TV 

A* as often as possible 1 

Bp about once each week 3 

C, occasionally 23 

D , rarely 31 

Ep never 24 



21. The use of "Talking Books" and the audio tapes was 



A, very helpful 5 

B, helpful 19 

C, adequate 37 

D, not very helpful 13 

E, useless 7 

No response CD 



22. Using the "Illustrated Book" is 

A. valuable 
B* helpful 

C , OK 

D . not very helpful 
E* wasted effort 
No response 

23. Could the Problem and Solution Book questions be answered without 
reading any references 



A. 


always 




2 


B, 


most of the time 




22 


C. 


sometimes 




42 


D. 


Seldom 




14 


E. 


never 


- ** - 


2 



6 

15 
35 

16 
7 

(3) 
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24. Did you study physics outside the classroom? 



A, over 6 hours per week 
R . 3—6 hours per week 

G. 1 — 3 h< urs per week 
D . seldom 
E - never 

25. Would specific review lectures by the instructor be helpful? 



A. 


very helpful 


43 


B . 


helpful 


26 


C . 


OK 


7 


D. 


wouldn T t be too helpful 


5 


E. 


not needed 


1 



11 

32 

33 
6 
0 



The next five questions refer to the following media: 



A. Text 

B. Instructor 

C. TV Tape 

D. Problem L jolution Book 

E. Talking Book/lllus trated Book 

26. Of the above five media which did you find most effective in 
learning physics? 



A. 

B . 

C. 

D . 

E, 

27. Which did you find second most effective? 

A, 

B. 

e. 

D. 

E. 

28. Which did you find third most effective? 



A. 

B. 

C. 

D. 

E. 



15 

22 

1 

40 

4 



29 

23 

1 

25 

4 



24 

25 
5 

11 

17 
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Which did you find fourth most effective? 



A. 10 

B . 9 

C. . 33 

D. ' 5 

E. 23 

No response (2) 

30* Which did you find least effective? 

A. 7 

B. 5 

C. 36 

D * 2 

E. , 29 

No response (3) 



31. When I do not know an answer to a question in the Problem and 
Solution Book , I most frequently 



A# 


make a random 


guess * 


17 


B# 


check my text 


or get help before 






starting . 




12 


C# 


try to pick the most reasonable 
answer based on dimensions and 





fT feel IT and other factors that I 
reason out * 45 

D * thumb thru and try to find the 
correct answer in the back of 
the book. • 8 

32. Based on my knowledge of my own habits and desires, the Problem 
and Solution Book: 

A. forces me to work more problems 

than I ordinarily would in a course 
like this * 46 

B . occupies the time I would rather 

be spending on problems from a 
textbook# 11 

C . lets me off the hook because X 
would normally spend more time 
working problems on my own, but 
1 feel X can quit when X finish 

the Problem and Solution Book# 25 

D. takes only part of the time I 
spend solving problems; I do 

extra problems from other sources • 0 
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3 . 



Not many students seem to use the talking book or illustrated book 
If you never or seldon use them, why is this so? 



A. They are very boring, 15 

B* They are too much trouble to use. 

C. They are not related to what X am 

tes ted on . 2 

D. They use odd notations and language 

that 'confuse me, 4 

E . I use them- 25 

No response (2) 

34. Do you learn more or lass from the TV tapes compared to a live 
lecture of the same length without a question period? 

A. much more 0 

B . somewhat more 8 

C. cJjout the same 24 

B, somewhat less 26 

E, much less 21 

No response (3) 

35# Do you feel an instructor is needed in the classroom? 

A, all the time 25 

B, most of the time 31 

C, sometimes 13 

D, rarely 11 

E* never 1 

No* response (1) 

36 * The requirement to make a minimum score on each Progress Check 

A. was good for me * because I knew I 

had to learn all the material 
eventually. 35 

B. was of no benefit to me because X 

didn* t study any harder, X just 
waited until I passed by check. 8 

G, interf erred with my learning because 

I was concerned about passing tests 
on past material* 11 

D. made no difference to me. 25 

No response (2) 
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APPENDIX E 



BACKGROUND VARIABLES 
AND RESPONSE STATISTICS 



Background variables, performance scores, and 
questionnaire responses by student* 

Correlation table, background variables and final 
exam score* 

Sel£=paced students summary statistics, background 
variables and final exam* 

Conventional students nummary statistics, background 
variables. 

Statistically significant ehi square* 
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SELF-PACED STUDENTS SUMMARY STATISTICS 
BACKGROUND VARIABLES AND FINAL EXAM 





SAT 

Verbal 


SAT 
Ma th 


Eng. 
Comp . 


COUNT 


82 


82 


82 


SUM 


48581,000 


54685.000 


48016.000 


SUMSQ 


29109024.000 


36771888.000 


28543200.000 


AVG 


592,451 


666.890 


585.561 


VAR 


4038,914 


3740.839 


5270,715 


STDEV 


63.552 


61.162 


72,600 


MAX 


' 749.000 


800.000 


762.000 


MIN 


466.000 


536.000 


447.000 


RANGE 


283,000 


264.000 


315,000 




Math 

Achiev. 


QPR 


Raw 

Score 

Physics 


COUNT 


82 


79 


72 


SUM 


54521.000 


22072.000 


1896.000 


SUMSQ 


36566176.000 


6384878.000 


54985.867 


AVG 


664.890 


279.392 


26.333 


VAR 


3987.284 


2796 590 


71.238 


STDEV 


62.428 


52,883 


8.440 


MAX 


800.000 


400.000 


44,000 


MIN 


510.000 


186.000 


10.000 


RANGE 


290.000 


214.000 


34.000 
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SELF-PACED STUDENTS SUMMARY STATISTICS 
BACKGROUND VARIABLES AND FINAL EXAM 
(Continued) 







Final 


Final 




Final 


Exam 


Exam 




Exam 


Sub. 1 


Sub , : 


COUNT 


82 


82 


82 


SUM 


2449,000 


1377.000 


1154.000 


3UMSQ 


76992,500 


24650,840 


17273.809 


AVG 


29,866 


16,793 


14.073 


VAR 


47.545 


18,856 


12,758 


STDEV 


6.895 


4.342 


3,572 


MAX 


47.000 


29,000 


23.000 


MIN 


17.000 


8.000 


7.000 


RANGE 


30.000 


21,000 


16.000 
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CONVENTIONAL STUDENTS SUMMARY STATISTICS 
BACKGROUND VARIABLES 
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The following pages present a compilation of 
chi square test results that were found to be statis- 
tically significant. 

The variable before the slash (/) is the primary 
sort variable used to select students for the statis- 
tical test; i,e* , low SAT V would indicate that those 
students whose SAT Verbal score was below average 
SAT V were included in the test. 

The first variable after the slash is the secondary 
sort variable. Students satisfying the first constraint 
were assigned to columns based on the secondary sort 
variable. If question 14 were the secondary sort vari- 
able , students who scored an A or a B were assigned to 
column 1, those who scored a C were assigned to 
column 2 S and those who scored a D or an E were 
assigned to column 3. When the secondary sort variable 
was a performance variable, column 1 corresponds to 
below average and column 2 corresponds to above average. 

The third sort variable defines the row assignment 
of each student. Where high, medium, and low did not 
constitute a valid breakdown (e.g., question 30), the 
matrix elements were not combined* 

Example : If we had low SAT V/question 14 x ques- 

tion 22, the first row in column 1 would be the number 
of students whose SAT V was low and who scored A or B 
for question 14 and A or B for question 22. 
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GROUP A: QUESTION 14 



All students/Ql4 x Q6 

18 4 6 

17 14 8 

4 2 9 

alpha ' .01 



High SAT M/Q14 x Q6 

8 2 2 

10 8 2 

2 0 5 

alpha *= .01 



All students/Ql4 x Q22 

12 . 1 8 

12 16 7 

12 .3 8 

alpha = ,01 



High SAT M/Q14 x Q22 

8 0 1 

5 7 3 

6 3 5 

alpha =*= , 05 



High Math aeh/ Q14 x Q6 

10 2 3 

9 7 1 

3 15 



alpha «= , 05 



Low SAT M/Q14 x Q22 

4 17 

7 9 4 

6 0 3 



alpha = .05 



High Math ach/Ql4 x Q22 

7 0 2 

6 8 2 

7 2 5 

alpha = . 05 
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GROUP A: QUESTION 14 (continued) 



High SAT 


V/Q14 x Q6 


Tow SAT V/Q14 


x Q22 


11 


3 1 


5 . 0 


6 


9 


5 5 


5 11 


3 


0 


1 5 


8 0 


3 


alpha = , 


01 


alpha = ,01 





High Eng 


Comp /Q 14 x Q6 


Low Eng Comp/Q14 


x Q22 


11 


3 


o 


3 0 


4 


10 


6 


3 


7 11 


6 


2 


0 


4 


4 0 


4 


alpha = , 


05 




alpha ■ , 05 





High 


QPR/Q14 x Q6 


High 


QPR/Q14 


x Q22 


6 


3 1 


4 


0 


4 


10 


6 2 


6 


8 


1 


2 


1 5 


7 


2 


3 


alpha 


* .05 


alpha 


-- .05 





Low raw 


phys/Q14 x Q6 


Low raw 


phys/ Q14 


x Q22 


8 


2 1 


4 


0 


3 


3 


7 5 


5 


10 


3 


1 


2 5 


3 


1 


5 


alpha - 


.05 


alpha = 


,05 
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GROUP lit QUESTION 24 



Low SAT 


V/Q 24 


x Q 30 


Low SAT 


M/Q 24 


x Q 30 


3 


0 


1 


3 


0 


6 


0 


0 


. 1 


1 


1 


1 


8 


11 


0 


11 


10 


0 


0 


1 


0 


1 


1 


0 


11 


4 


1 


8 


4 


0 


alpha » 


. 01 




alpha = 


, 05 





Low Eiig Comp /Q 24 x Q 30 

3 0 1 

111 
8 10 0 

0 10 
11 2 0 

alpha ,01 



Low QPR/Q 24 x Q 30 



0 0 1 

0 11 

11 12 0 

0 10 

10 3 1 



alpha — ,01 



Low Hath ach/Q24 x Q26 



4 3 1 

8 7 0 

1 -0 0 

11 5 0 

0 0 1 



alpha = ,01 
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GROUP Ct MEDIA RELATED SUBTEST 



All students /media related x Q30 Low SAT M/media related x Q30 



4 


3 


1 . 


2 


0 


5 


0 


3 


26 


10 


17 


4 


1 


1 


1 


1 


14 


15 


6 


6 


alpha = 


. 05 


alpha = . 05 





Low raw phys/media related x Q26 

4 
C 
1 
8 
0 

alpha = .05 



1 

10 

0 

10 

0 
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FINAL EXAM 



GROUP E ; 



Low Eng Comp/ final x Q30 



High raw phys /final x Q30 



3 1 

0 3 

15 3 

1 0 

6 7 



1 

0 

12 

0 

3 



3 

3 

4 
0 

10 



alpha = , 05 



alpha “ , 05 
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GROUP E: 



CORE PROBLEM SUBTEST 



All students/core x Q30 



Low SAT M/core x Q30 



r 

1 

28 

1 

13 



2 

4 

8 

1 

16 



3 

0 

18 

1 

5 



alpha == . 05 



alpha = .01 



Low Eng Comp /core x Q30 

4 

0 

16 

1 

8 

alpha - .01 



0 

3 

2 

0 

5 
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